Framing the Problem¶
An exoplanet is any planet beyond our solar system¶
Its not a simple process to identify or catalog exoplanets. Even with high powered scopes, there has not been many exoplanet observed directly. Most of them are detected indirectly via astronomical data.
We want to see if we have the ability to confirm planet's dispositionwith other data not typically used by NASA.
Why we care¶
Exoplanet Exploration is important because it is the process in which we can start to identify "Earth-like" planets. From identifying these planets, we can start to search for habitable planets and for life outside of our solar system. This also leads into the planning of future missions as our knowledge of space evolves.
How Success will be measured¶
We determine success by the accuracy of our model which is the hitrate of the Random Forest of variables not typically used by NASA to build it.
Assumptions:¶
Decision Tree and Random Forest¶
No formal assumptions are required for Decision Tree and Random Forest
Logistic Regression¶
1.) Observations are independent of each other
2.) Little or No Multicollinearity among independent variables
3.) Independent Variables are linearly related to the log of odds
4.) Dependent Variable is binary
5.) Large Amount of Observations
Data¶
NASA Kepler Objects of Interest Data Set
A Kepler object of interest (KOI) is a star observed by the Kepler space telescope that is suspected of hosting one or more transiting planets
#load data
data = pd.read_csv('cumulative_2020.07.07_13.36.50.csv')
data.head()
data.info()
Wrangle Data¶
#make dummy variable for our target
data = pd.get_dummies(data, columns = ['koi_disposition'])
data = data.rename(columns ={"koi_disposition_CONFIRMED":"Confirmed"})
data = data.rename(columns ={"koi_disposition_CANDIDATE":"Candidate"})
data.head()
#remove candidate rows
dat = data[data['Candidate']==0]
dat.shape
#look at % missing null values
dat.isnull().sum()/len(dat)*100
# use numeric data not pertaining to flag, err or score
# the above varaibles are the only NASA used to determain 'Confirmed'
# we want to see if we can determine confirmed with the other data
# remove columns with missing values over 20%
df = dat[['Confirmed','koi_kepmag','dec','ra','koi_srad','koi_slogg','koi_steff','koi_model_snr','koi_insol','koi_teq','koi_prad','koi_depth','koi_duration','koi_impact','koi_time0bk','koi_period']]
df.head()
df.info()
#replace null values
for i in df.columns :
if type(df[i][1]) == np.int64 :
avg = df[i].astype('int').mean(axis=0)
df[i].replace(np.nan, avg, inplace=True)
elif type(df[i][1]) == np.float64:
avg = df[i].astype('float').mean(axis=0)
df[i].replace(np.nan, avg, inplace=True)
else:
pass
df.info()
Examine Variables¶
#look at correlations
d = df.drop(columns='Confirmed')
cor = d.corr()
cor
var
# Correlated:
#koi_srad, koi_insol
#koi_model_snr, koi_depth
#koi_prad, koi_impact
#koi_period, koi_time0bk
Training and Test Sets¶
#make training sets
X_train, X_test, y_train, y_test = train_test_split(d, target, test_size=0.2, random_state=9)
#do classifier to find feature importance
model = RandomForestClassifier(random_state=1, max_depth=10)
model.fit(X_train,y_train)
#plot feature importance
features = X_train.columns
importances = model.feature_importances_
indices = np.argsort(importances)[-15:] # top 10 features
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [features[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
Modeling¶
#Make Decision Trees
tre3 = tree.DecisionTreeClassifier(max_depth=7)
tre3.fit(X_train,y_train)
ypred3 = tre3.predict(X_test)
print("Accuracy:", metrics.accuracy_score(y_test, ypred3))
#Defining the feature names
df_exo_x = ['koi_time0bk', 'koi_impact', 'koi_duration', 'koi_depth', 'koi_prad', 'koi_teq', 'koi_insol',
'koi_period', 'koi_model_snr', 'koi_steff', 'koi_slogg', 'koi_srad', 'ra', 'dec', 'koi_kepmag']
#Decision Tree Visualization
dot_data = StringIO()
export_graphviz(tre3, out_file=dot_data,
filled=True, rounded=True,
special_characters=True, feature_names = df_exo_x, class_names=['0', '1'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png('disposition2.png')
Image(graph.create_png())
print("Accuracy:", metrics.accuracy_score(y_test, ypred4))
dot_data = StringIO()
export_graphviz(tre4, out_file=dot_data,
filled=True, rounded=True,
special_characters=True, feature_names = df_exo_x, class_names=['0', '1'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png('disposition2.png')
Image(graph.create_png())
#look at confusion matrices and hitrates
cm = confusion_matrix(y_test,ypred3)
hr_dt = (cm[0,0]+cm[1,1])/1430
mse_dt = mean_squared_error(y_test, ypred3)
print(cm)
print('HitRate:', (cm[0,0]+cm[1,1])/1430)
cm2 = confusion_matrix(y_test,ypred4)
print(cm2)
print('HitRate:', (cm2[0,0]+cm2[1,1])/1430)
# Look at accuracies depending on depth for Random Forest
# Setup arrays to store train and test accuracies
dep = np.arange(1, 20)
train_accuracy = np.empty(len(dep))
test_accuracy = np.empty(len(dep))
# Loop over different values of k
for i, k in enumerate(dep):
# Setup a k-NN Classifier with k neighbors: knn
clf = RandomForestClassifier(max_depth=k)
# Fit the classifier to the training data
clf.fit(X_train, y_train)
#Compute accuracy on the training set
train_accuracy[i] = clf.score(X_train, y_train)
#Compute accuracy on the testing set
test_accuracy[i] = clf.score(X_test, y_test)
# Generate plot
plt.title('clf: Varying depth of tree')
plt.plot(dep, test_accuracy, label = 'Testing Accuracy')
plt.plot(dep, train_accuracy, label = 'Training Accuracy')
plt.legend()
plt.xlabel('Depth of tree')
plt.ylabel('Accuracy')
plt.show()
#random forest models
model = RandomForestClassifier(n_estimators = 100,max_depth=13, bootstrap=True)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
# The mean squared error
mse = mean_squared_error(y_test, y_pred)
print('Mean squared error:', mse)
# The coefficient of determination: 1 is perfect prediction
r2 = r2_score(y_test, y_pred)
print('R-Squared:', r2)
cm= confusion_matrix(y_test,y_pred)
print(cm)
print('HitRate:', (cm[0,0]+cm[1,1])/1430)
model = RandomForestClassifier(n_estimators = 100,max_depth=8, bootstrap=True)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
# The mean squared error
mse_rf = mean_squared_error(y_test, y_pred)
print('Mean squared error:', mse)
# The coefficient of determination: 1 is perfect prediction
r2 = r2_score(y_test, y_pred)
print('R-Squared:', r2)
cm= confusion_matrix(y_test,y_pred)
hr_rf = (cm[0,0]+cm[1,1])/1430
print(cm)
print('HitRate:', (cm[0,0]+cm[1,1])/1430)
#logistic regression models
model = LogisticRegression().fit(preprocessing.scale(X_train),y_train)
y_pred = model.predict(preprocessing.scale(X_test))
mse_log = mean_squared_error(y_test, y_pred)
print('Mean squared error:', mse)
# The coefficient of determination: 1 is perfect prediction
r2 = r2_score(y_test, y_pred)
print('R-Squared:', r2)
cm= confusion_matrix(y_test,y_pred)
hr_log = (cm[0,0]+cm[1,1])/1430
print(cm)
print('HitRate:', (cm[0,0]+cm[1,1])/1430)
Compare Models¶
sns.catplot(x ='mod', y = 'HitRate', data = models, kind = 'bar')
sns.catplot(x ='mod', y = 'mse', data = models, kind = 'bar')
Fine Tuning¶
#Take out low varance varables
df2 = pd.concat([target,df[variable]],axis = 1)
d2 = df2.drop(columns='Confirmed')
X_train2, X_test2, y_train2, y_test2 = train_test_split(d2, df2['Confirmed'], test_size=0.2, random_state=9)
model = RandomForestClassifier(n_estimators = 100,max_depth=8, bootstrap=True)
model.fit(X_train2, y_train2)
y_pred = model.predict(X_test2)
# The mean squared error
mse = mean_squared_error(y_test2, y_pred)
print('Mean squared error:', mse)
# The coefficient of determination: 1 is perfect prediction
r2 = r2_score(y_test2, y_pred)
print('R-Squared:', r2)
cm= confusion_matrix(y_test2,y_pred)
print(cm)
print('HitRate:', (cm[0,0]+cm[1,1])/1430)
Doesn't effect hitrate but reduces dimensionality and mse!¶
#remove multicollinearity:
#koi_srad, koi_insol 0.579
#koi_model_snr, koi_depth 0.56
#koi_prad, koi_impact 0.6798
d2.corr()
#plot feature importance
features = X_train2.columns
importances = model.feature_importances_
indices = np.argsort(importances)[-15:] # top 10 features
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [features[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
#remove koi_impact because it has highest collinearity and lower feature
#importance than koi_prad
df3 = df2.drop(columns = 'koi_impact')
d3 = df3.drop(columns='Confirmed')
X_train3, X_test3, y_train3, y_test3 = train_test_split(d3, df3['Confirmed'], test_size=0.2, random_state=9)
Final Model¶
model_final = RandomForestClassifier(n_estimators = 100,max_depth=8, bootstrap=True)
model_final.fit(X_train3, y_train3)
y_pred = model_final.predict(X_test3)
# The mean squared error
mse = mean_squared_error(y_test3, y_pred)
print('Mean squared error:', mse)
# The coefficient of determination: 1 is perfect prediction
r2 = r2_score(y_test3, y_pred)
print('R-Squared:', r2)
cm= confusion_matrix(y_test3,y_pred)
print(cm)
print('HitRate:', (cm[0,0]+cm[1,1])/1430)
Reduced HitRate a little, but worth it to reduce multicollinearity¶
plot_confusion_matrix(cm, classes=["Confirmed", "Unconfirmed"], cmap=plt.cm.Greys)
plt.show()
estimator = model_final.estimators_[1]
#from sklearn.tree import export_graphviz
# Export as dot file
export_graphviz(estimator, out_file='tree.dot',
feature_names = X_test3.columns,
class_names = ['0','1'],
rounded = True, proportion = False,
precision = 2, filled = True)
# Convert to png using system command (requires Graphviz)
from subprocess import call
call(['dot', '-Tpng', 'tree.dot', '-o', 'tree.png', '-Gdpi=600'])
# Display in jupyter notebook
from IPython.display import Image
Image(filename = 'tree.png')
Final Variables¶
12 Variables In the Final Model¶
1.) koi_kepmag - K-band mag from 2MASS - basically a complex formula used to determine or help determine planets flux
2.) dec - KIC Declination
3.) ra - KIC Right Ascension
4.) koi_srad - The photospheric radius of the star
5.) koi_steff - The photospheric temperature of the star.
6.) koi_model_snr - Transit depth normalized by the mean uncertainty in the flux during the transits. (the energy a planet receives per second)
7.) koi_insol -Insolation flux is another way to give the equilibrium temperature.
8.) koi_teq - Approximation for the temperature of the planet.
9.) koi_prad - The radius of the planet. Planetary radius is the product of the planet star radius ratio and the stellar radius.
10.) koi_depth - The fraction of stellar flux lost at the minimum of the planetary transit.
11.) koi_duration - The duration of the observed transits.
12.) koi_time0bk - The time corresponding to the center of the first detected transit in Barycentric Julian Day (BJD) minus a constant offset of 2,454,833.0 days
Interesting Facts / Lessons Learned¶
Assumptions¶
Thought there would be more multicolinearity
What did work¶
Random forest worked
Removing low variance variables
Decent model after removing flag and error variables
What didn't work¶
Logistic Regression and Decision Tree when compared to Random Forest
A lot of debugging issues involving packages
Hard to graph raw data because of large amount of variables
Surprises¶
Flux or the energy the planet receives from their star per second, was an important variable
Surprised on how well Random Forest works for the model
References¶
exoplanets.nasa.gov/exep/
exoplanetarchive.ipac.caltech.edu/docs/API_kepcandidate_columns.html#pdisposition